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(57) Abstract 

A virtual reality medical simulation system (1) provides a highly realistic simulation of a variety of medical procedures, particularly 
catheter based procedures such as balloon angioplasty. A catheter interface device (3) tracks a catheter wire (7) in translation and rotation 
and outputs a signal to a computer program that displays movement of a virtual catheter within a virtual arterial tree including an occluded 
region. The catheter tip is navigated through the arterial tree and into the occluded region, whereupon the computer program generates a 
tactile feedback force that is output to the catheter interface device (3) to simulate increased resistance. Once the catheter (7) is properly 
positioned within the occlusion, a virtual balloon catheter can be inflated to remove the occlusion. A variety of special effect enhancements 
arc included in the simulation, including 3-D imaging, simulated radiopaque dye infusion, bleeding and atrial pulsation in synchronism with 
an on screen EKG display. 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international 
applications under the PCT. 



AM 


Armenia 


GB 


United Kingdom 


MW 


Malawi 


AT 


Austria 


GE 


Georgia 


MX 


Mexico 


AU 


Australia 


GN 


Guinea 


NE 


Niger 


BB 


Barbados 


GR 


Greece 


NL 


Netherlands 


BE 


Belgium 


HU 


Hungary 


NO 


Norway 


BF 


Burkina Faso 


IE 


Ireland 


NZ 


New Zealand 


BG 


Bulgaria 


IT 


Italy 


PL 


Poland 


BJ 


Benin 


JP 


Japan 


FT 


Portugal 


BR 


Brazil 


KE 


Kenya 


RO 


Romania 


BY 


Belarus 


KG 


Kyrgyscan 


RU 


Russian Federation 


CA 


Canada 


KP 


Dernocratic People's Republic 


SD 


Sudan 


CF 


Central African Republic 




of Korea 


SE 


Sweden 


CG 


Congo 


KR 


Republic of Korea 


SG 


Singapore 


CH 


Switzerland 


KZ 


Kazakhstan 


SI 


Slovenia 


CI 


Cote d'lvoire 


LI 


Liechtenstein 


SK 


Slovakia 


CM 


Cameroon 


LK 


Sri Lanka 


SN 


Senegal 


CN 


China 


LR 


Liberia 


sz 


Swaziland 


CS 


Czechoslovakia 


LT 


Lithuania 


TD 


Chad 


CZ 


Czech Republic 


LU 


Luxembourg 


TG 


Togo 


DE 


Germany 


LV 


Latvia 


TJ 


Tajikistan 


DK 


Denmark 


MC 


Monaco 


TT 


Trinidad and Tobago 


EE 


Estonia 


MD 


Republic of Moldova 


UA 


Ukraine 


ES 


Spain 


MG 


Madagascar 


UG 


Uganda 


FI 


Finland 


ML 


Mali 


US 


United States of America 


FR 


France 


MN 


Mongolia 


UZ 


Uzbekistan 


GA 


Gabon 


MR 


Mauritania 


VN 


Viet Nam 



WO 96/28800 



PCT/US96/02317 



COMPUTER BASED MEDICAL PROCEDURE SIMULATION SYSTEM 

BACKGROU ND OF THE INVENTION 

The present invention relates to computer-based simulation of medical procedures, 
e.g., surgical procedures. A particular application of the present invention concerns a 
system for providing a realistic "virtual reality" simulation of a cardiac catheterization 
procedure. 

Continuing public debate concerning rising health care costs has pin-pointed 
continuing medical education (CME) and graduate medical education (GME) as major 
expenses to address in the health reform effort. More than $6 billion is spent on CME and 
GME annually. Much of this cost is attributed to expensive training, resources and 
material. The use of virtual reality (VR) in medical education and training can reduce the 
expenses associated with changes in medical technology, as well as provide enhanced 
physician training. 

VR technology has been used in a wide variety of fields, including military, 
architecture, chemistry and biomedicine. The application of VR technology to education 
and training, however, is just beginning to be exploited. VR technologies are becoming 
more applicable in medicine, in part because of the high costs of traditional training 
resources (lab animals, physican's time, etc.). 

The value of computer-assisted instruction has long been recognized. During the 
past decade, numerous medical centers around the country have designed and used software 
for medical education. The software in use is primarily designed to run on IBM PC 
compatible and Macintosh computers. These instructional programs also have many 
advantages over traditional lecture-oriented learning, e.g., they are interactive and allow 
for self-paced instruction. Many of the programs also incorporate simulations which 
require the user to make patient management decisions. If incorrect decisions are made, 
text describing why the choice was incorrect is provided. 

More recently, medical education programs have been created using personal 
computer graphics for the instruction of anatomy. This type of program provides increased 
educational value because it gives students the opportunity to quickly access anatomic 
images and related textual data. Rather than spending hours in the gross anatomy lab 
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dissecting cadavers (and being exposed to toxic chemical preservatives), these programs 
allow students to work in the library — or at home — and spend as much time as needed to 
learn the material. The programs also provide textbook references so students can 
immediately read about the clinically relevant aspects. 

Although these computer-assisted programs offer many advantages in medical 
education, there are some inherent limitations and drawbacks. For example, these 
programs are confined to two dimensional (2-D) imaging. This limits the scope of 
depicting anatomical orientation, particularly with respect to showing the relation of 
superficial and deeper structures. These programs are similarly limited by lacking the 
ability to rotate in space so that anterior and posterior relationships can be seen. 
Additionally, while highly valuable for anatomic instruction, these programs do not allow 
for practice of actual medical procedures under realistic conditions. 

A system capable of providing a realistic real-time simulation of surgical procedures 
would help reduce the operative risk associated with new technologies in medicine, allowing 
better transference of skills from training to the operating room. A tremendous demand 
exists for enhancing the way that physicians learn new invasive procedures. Appropriate 
education in new medical and surgical procedures is often outpaced by the desire of 
physicians to incorporate a procedure into their practice. 

As one example, until a few years ago, the treatment for coronary artery disease was 
open heart surgery, an expensive and very difficult surgical procedure. As an alternative 
to open heart surgery, percutaneous transluminal coronary angioplasty (PTCA) is now 
generally considered safe and effective to relieve the same conditions that heart surgery did 
just a few years ago. With the recent availability of this new procedure, the training of 
physicians presents a considerable problem. Currently, physician experience depends on 
the availability of a weekend course and practicing the procedure in an animal lab. 
Unfortunately, two days rarely provides enough practice time. Additionally, animal 
anatomy is quite different than human anatomy. A realistic virtual reality simulation could 
offer the physician the ability to practice hundreds of procedures in a realistic environment, 
and with realistic anatomy, prior to patient contact. 

Computerized surgical simulations have the potential for improving surgical 
morbidity and mortality. Studies have shown that for a wide range of diagnostic and 
therapeutic procedures, doctors doing their first few to several dozen cases are much more 
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likely to make a greater number of errors. This phenomenon has been referred to as the 
"learning curve. * Adequate proctoring of learners by experienced surgeons is cumbersome, 
as there are few surgeons experienced enough to proctor their colleagues. It is difficult for 
physicians, particularly those in rural areas, to travel to larger medical centers for training. 
The requirement also places a burden on experts who, already busy with their clinical 
responsibilities, could become overwhelmed with proctoring requests. 

Another potential advantage of VR surgical training is in the standardization of 
surgical training. Such standardization would help insure, e.g., that all board-certified 
physicians are top-notch. Providing training with computer programming would allow 
major medical centers to directly interact with students and physicians in a rural setting. 
By so doing, the credentialing and granting of privileges for traditional procedures, as well 
as newer procedures like laparoscopic surgery, could be objectively provided based on 
standardized skills assessment, regardless of where a surgeon is located. The challenge has 
been to create a surgical simulation that is sufficiently realistic that the motor skills and 
anatomic recognition acquired in the simulation are readily transferred to performance of 
the actual procedure. 

In addition to realistically simulating the involved anatomy, i.e., body parts, a 
realistic computer-based surgical simulator must also provide a realistic surgical 
environment. A key in this regard is to provide the involved proxy medical instrument(s) 
with a realistic look and feel. This can be accomplished through the use of computer 
peripheral devices that mimic the actual instrument(s) used by a surgeon to perform an 
actual procedure. In this respect, the closer the proxy surgical instrument is to the actual 
surgical instrument, the more realistic the surgical simulation. 

One computer interface device that has been used in surgical simulations is the 
Immersion PROBE produced by Immersion Corporation of Palo Alto, California. The 
Immersion PROBE comprises a pen-like stylus supported on a light-weight mechanical 
linkage. The stylus is held between the fingers like a medical instrument and is wielded 
dexterously. The Immersion PROBE reports the position and orientation of the stylus to 
a host computer via a standard serial port interface. The Immersion PROBE linkages 
provide six degrees of freedom. Sensors provided at the linkage joints convey spatial 
coordinates (X,Y,Z) and orientation (roll, pitch, yaw) of the stylus to the host computer. 

While the Immersion PROBE is a useful general purpose 3-D computer/user 
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interface tool, it does not closely resemble in feel or appearance an actual surgical or 
medical instrument. For example, it's pen-like stylus and six degrees of freedom bear little 
resemblance to the type of elongated flexible member used in a catheterization procedure. 
An additional shortcoming of the Immersion PROBE is that it has no provision for 
providing tactile feedback to the user. Tactile feedback is very important to create a 
realistic feel. 

Hon U.S. Patent No. 4,907,973 discloses a computer-based surgical simulation 
system including a peripheral device designed to simulate a surgical instrument, as well as 
the involved anatomy, i.e. , body parts. In particular, the Hon patent discloses a system for 
computer simulation of a balloon-angioplasty operation. Therein, a catheter-like device is 
negotiated through a mock arterial path of a physical internal body part modeling device. 
Sensors on the catheter track the progress of the catheter through the mock arterial path. 
In one embodiment, a vessel constricting simulator within the mock arterial path provides 
a fixed resistance to the progress of the catheter. 

While the Hon system has the potential for providing a more realistic feel of a 
catheter device as compared with the Immersion PROBE, it lacks flexibility for easily 
altering the involved anatomy and procedures. The simulation shown on the computer 
monitor appears to correspond to the structure of the physical model. Providing different 
anatomies, patients conditions and procedures would require additional physical models for 
each. 

Another shortcoming of the Hon system is its lack of "special effects* for 
realistically simulating changes in the virtual environment responsive to operator 
manipulations and performance. For example, in the catheter insertion simulation of Hon, 
no provision is made for (1) a realistic simulation of bleeding, e.g., in the event an arterial 
wall is punctured, (2) radio-opaque dye infusion to render the arteries visible under a 
fluoroscope, or (3) catheter balloon inflation and consequent removal of an inclusion. 
Moreover, the Hon system is over-simplified and less than realistic, since no provision is 
made for tracking and displaying catheter rotation in the virtual environment. Importantly, 
in an actual catheterization procedure, rotation of the catheter wire is used, e.g., to direct 
the catheter in the proper direction. 
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SUMMARY O F THE INVENTION 

In view of the foregoing, a principal object of the present invention is to provide for 
the performance of a simulated medical procedure under highly realistic conditions. 

It is a further object of the invention to provide a high degree of flexibility, thereby 
allowing performance of a variety of medical procedures under different patient conditions, 
including actual patient conditions. 

Yet another object of the present invention is to combine with a realistic simulation 
experience, tutorial aspects such as expert assistance and evaluation. 

These and other objects are achieved by a computer based medical procedure 
simulation system in accordance with the present invention. In one such system, first 
memory means are provided for storing image data representative of a virtual internal body 
environment. Second memory means are provided for storing image data representing a 
virtual medical instrument extending within the internal body environment. Display means 
are provided for displaying images corresponding to the image data stored in the first and 
second memory means. A first computer peripheral device tracks the movement of a 
physical member representing the virtual medical instrument, and outputs a signal based 
thereon. First input means are provided for receiving the signal from the peripheral device. 
First calculating means are provided for calculating position data representing the position 
of the virtual medical instrument within the virtual internal body environment, based on the 
signal from the peripheral device. First updating means are provided for updating the 
image data representing the virtual medical instrument extending within the virtual internal 
body environment, based on said calculated position data. 

An exemplary embodiment of the present invention is a cardiac catheterization 
simulator used to train interventional cardiologists to perform specific catheterization 
procedures such as angioplasty and stent placement. The cardiologist works with the 
system in the same way that he/she would perform an actual catheterization, i.e., by 
manipulating a proxy catheter. The location and movement of the catheter is monitored by 
watching a simulated fluoroscopic view, and the cardiologist can feel the interaction 
between the catheter and the artery by the force required to move the catheter. 

These and other objects, features and advantages of the present invention will be 
readily apparent and fully understood from the following detailed description of preferred 
embodiments, taken in connection with the appended drawings. 
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RRT ^F DESCRIPTION OF THE DRAWINGS 

Figure 1 is a schematic view of an overall medical procedure simulation system in 
accordance with the present invention. 

Figure 2 is a schematic view illustrating the operating principles of a catheter 
interface device used in the system shown in Fig. 1. 

Figures 3-17 are logic flow diagrams for a program used in the simulation system 
of Fig. 1. 

Figures 18-29 are representative screen displays produced in the simulation system 
of the present invention. 

Figure 30 is a front-left perspective view of the catheter interface device used in the 
simulation system shown in Fig. 1. 

Figure 31 is a rear-right perspective view of the catheter interface device shown in 
Fig. 30, with the cover removed. 

Figure 32 is a front elevational view of the internal componentry of the catheter 
interface device, with the electrical connections omitted for visual clarity. 

Figure 33 is a rear elevational view of the internal componentry of the catheter 
interface device, with the electrical connections omitted for visual clarity. 

Figure 34 is a right-side elevational view of the internal componentry of the catheter 
interface device, with the electrical connections omitted for visual clarity. 

Figure 35 is a left-side elevational view of the internal componentry of the catheter 
interface device, with the electrical connections omitted for visual clarity. 

Figure 36 is a top-plan view of the internal componentry of the catheter interface 

device. 

Figure 37 is a cross-sectional view taken along a line 37-37 in Fig. 36. 
DFTAnm DESCRIPTIO N OF THE PREFERRED EMBODIMENTS 

An overall simulation system 1 in accordance with the present invention is illustrated 
in Fig. 1. The system comprises two primary parts: a computer/human interface device 
3 and a computer 5 storing a software program which works in conjunction with the device. 
In the exemplary embodiment described in detail herein, the system simulates a cardiac 
catheterization procedure, and interface device 3 is a catheter interface device. Catheter 
interface device 3 tracks the translational and rotational movement of a proxy catheter 7 
provided in the form of a wire or like elongated flexible object. Catheter interface device 
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3 also provides selective tactile feedback resistance to the motion of the proxy catheter. 
The translation and rotation of the wire are converted into digital signals which are sent 
over line 9 to the simulation program. The virtual reality simulation is presented on a high 
resolution computer monitor 10. The feedback resistance is controlled by digital signals 
11 sent by the program back to the catheter interface device over line 11. 

In addition to the catheter interface device, the system comprises a switching 
mechanism, e.g., an ON-OFF foot switch 13. When activated in one mode, the switch 
triggers the software to simulate a release of radio-opaque contrast material. This allows 
for visualization of the coronary arteries, including the area in the artery that is obstructed, 
as in an actual procedure. In a second mode encountered when the virtual catheter enters 
an occluded portion of the virtual artery, the switch serves to simulate a syringe operated 
to selectively inflate and deflate a virtual balloon catheter. Randomly generated or user 
selected patient determinative data will determine how many successive inflations and 
deflations of the virtual balloon catheter are required to successfully remove the occlusion. 
In place of a simple footswitch providing a dual function, separate devices (e.g., proxy 
syringes) may be employed to actuate dye infusion and balloon inflation. Other devices that 
may be incorporated into the various types of catheterization procedures may be employed 
as well. 

For enhanced effect, the screen simulations can be presented in 3-D. This is 
accomplished through the use of a stereographic system, e.g., CrystalEyes available from 
Stereo Graphics, Inc. of San Rafael, California. This product uses eyeglasses 15 having 
active liquid crystal shutters which open and close sixty times per second. The shutters are 
synchronized to monitor 10 via an infrared emitter 17 mounted on the front of the monitor, 
and a receiver 19 mounted on the glasses. Although not used in an actual angioplasty 
procedure, 3-D imagery provides a more sophisticated examination of the cardiac anatomy 
for these types of procedures. A surgeon's ability to see organs and tissues in 3-D, in 
laparoscopic procedures, and similar angioplasty procedures, will give physicians an 
enhanced ability to view and manipulate those organs and tissues. 

Digital, compact-disc quality multi-media audio can be used to provide an 
instructional narration in the inventive system, as well as to provide soothing background 
music as is often provided during actual surgery. For this purpose, the illustrated system 
includes a CD-ROM drive 21, an audio mixer 23 (for mixing background music from the 
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CD-ROM drive and a narrative "voice-over" stored on the computer harddrive), and a pair 
of amplified speakers 25. 

The operating principles of the catheter interface device tracking mechanism are 
schematically illustrated in Fig. 2. To obtain translation tracking, a friction wheel 27 
presses against the catheter wire 7. This causes the wheel to rotate when the wire is pushed 
or pulled. 

An optical encoder 29 is attached to the friction wheel for measuring the rotation. 
Rotation measurement is obtained by passing the catheter wire through a semi-tight 
grommet 31. When the wire is rotated, a wheel 33 attached to grommet 31 rotates. An 
optical sensor 35 mounted adjacent to the wheel measures the rotation of wheel 33 and 
outputs a corresponding signal. At the same time, grommet 31 allows the catheter wire to 
slide translationally therethrough. 

A feedback resistance force is achieved by having wire 7 slide through a wire 
pressing mechanism 37 actuated by a solenoid 39. When the solenoid is activated, it pushes 
a plunger 41 against one side of a sleeve 43, pressing the catheter wire 7 and thereby 
making it more difficult to move the same. 

An exemplary catheter based simulation is the simulation of PTC A. As mentioned 
above, PTCA is a technique for relieving partial blockage of arteries in the heart. In the 
actual procedure, a wire-like guiding catheter is threaded through an artery in the leg, up 
to the blocked area in the heart. The surgeon can see where the wire is going by injecting 
radio-opaque contrast material into the artery, thereby rendering the artery visible on a 
fluoroscopic (X-ray) display. The physician directs the catheter's placement by twisting and 
pushing as it is guided through the artery. Once the end of the catheter is in the proper 
position, the surgeon can thread a balloon catheter onto the guiding catheter. Once the 
balloon catheter is properly placed, the surgeon can inflate the balloon to open the occluded 
portion of the artery so that regular blood flow is restored. More recently, a self-guiding 
balloon catheter have been used to perform the procedure, thus eliminating the preliminary 
step of placing a separate guiding catheter. 

The exemplary simulation system shown and described in detail herein proceeds very 
much like an actual PTCA procedure, particularly one using a self-guiding balloon 
catheter. Additionally, a number of simulation enhancements are provided, as described 
in detail herein below. It will be appreciated that the principles of the invention could be 
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applied in order to create a double catheter insertion simulating more closely the more 
conventional technique of first inserting a guiding catheter and then inserting a separate 
balloon catheter thereover. 

The software monitors the translation and rotation of catheter wire, as the 
cardiologist manipulates it. Using this information, the program tracks the progress of the 
catheter through a simulated arterial tree forming part of a 3-dimensional internal body 
environment. The simulation program uses geometry data which can be created for each 
specific procedure (and virtual patient). The location and orientation of catheter 7 is 
converted into an image by the program, which image is displayed on computer monitor 
10. The image has the same fluoroscopic appearance as a cardiologist sees during an actual 
procedure (but with the optional enhancement of 3-D imaging) (see the representative screen 
displays of Figs. 22-26). In addition, other views which are not possible during a 
procedure can be provided by the simulation program to enhance the instructional value of 
the simulation (see Figs. 27-29). Depending on the position and orientation of the virtual 
catheter within the virtual arterial tree, the simulation program may calculate and output 
to catheter interface device 3 a resistance force signal so that the cardiologist can experience 
the feel of the catheter insertion, just as he/she would during the actual procedure. In 
particular, in the preferred embodiment, a tactile feedback force is generated when the 
virtual catheter enters an occluded portion of the virtual artery. 

The system can add additional realism by displaying at the outset patient history and 
laboratory data screens (see Figs. 18-20). The system can also incorporate variations in 
complications corresponding to different patient conditions, the presentation of which can 
be randomly generated by the computer, or selected by the user. 

In order to provide additional realism, the system can simulate procedural error 
conditions. For example, if the catheter wire is moved too abruptly, this triggers the 
simulation of a puncture in the arterial wall, and associated bleeding (see Fig. 25). If the 
virtual balloon catheter is inflated to an excessive amount, a realistic simulation of balloon 
rupture occurs. 

As seen in Figs. 21-29, a simulated electrocardiogram (EKG) waveform can also 
be displayed on the monitor 10. Provision is made for changing the waveform from 
"abnormal" to "normal" during the course of the simulation. Provision can also be made 
for the random occurrence of abnormalities such as atrial fibrillation. Additional realism 
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is provided by simulating pulsation of the arterial tree (and the catheter therein) in 
synchronism with the EKG. 

Additional abilities that can be provided by the simulation system include, among 
others, interactive control of the point of view, recording of the procedure, and monitoring 
by an expert assistant. Changing the point of view is important. For example, viewing the 
arterial tree from different perspectives allows the cardiologist or student to better learn the 
3-dimensional relationships of the arterial branches. This is difficult to master in practice 
since only a 2-dimensional projection of the 3-dimensional geometry is available in the 
fluoroscopic view used during an actual angioplasty procedure. The progress of the 
cardiologist can be recorded and played back for review and evaluation. A virtual "expert 
assistant" can be included to monitor the actions of the cardiologist during the simulation 
and can provide information such as warnings and explanations. 

Many different catheter-based procedures can be simulated with the inventive 
system. The specific procedure is created by the data used in the interaction aspects of the 
simulation program. Thus, the same catheter interface device can be used with many 
different procedures while the simulation program is customized for each procedure. This 
enables the catheterization simulation system to be used to create training environments for 
many different clinical procedures and patient conditions. The patient conditions can be 
simulated or actual. With respect to actual conditions, those conditions could be taken from 
representative past patients for use in general training, or could be taken from a current 
patient for use in a pre-operative surgical rehearsal. 

The logic flow of the exemplary PTCA simulation program (and representative 
screen displays thereof) are now described in detail, with reference to Figs. 3-29. 

Figure 3 depicts three primary modules of the main program: error checking 45, 
case presentation 47 and calibration 49. In error checking module 45, features and function 
of the system are diagnosed for presence and functionality. As previously mentioned, a 
stereographic 3-D viewing system can be utilized by the system to enhance the simulation. 
To determine if the stereographic transmitter is present, a call to the device is made at step 
49. If the transmitter is present, the video is adjusted for three dimensional perception at 
step 51. If the transmitter is not present, the stereographic viewing function is disabled at 
step 53 and the program proceeds without the visual enhancement. The system may 
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alternatively utilize other types of visual perception enhancement devices, such as 
headmounted displays, and projection devices (including holographic projection devices). 

As previously mentioned, the system utilizes a catheter interface device 3 that 
provides for direct human interaction with the computer simulation. The presence and 
functionality of catheter interface device 7 is determined at step 55. If the device is not 
detected or if it fails a diagnostic routine, the error is reported via the display and the 
simulator program will exit to the host operating system (step 57). 

Given that catheterization interface device 3 passes the previous error checking 
routines, the program next proceeds to the detection of footswitch 13 used for dye infusion 
and balloon inflation (step 59). If the system fails to detect the footswitch, an error is 
visually reported and the program exits to the host operating system (step 61). 

To synchronize the program polling rate, simulation timing functions, and threshold 
arrays, the program implements a routine to assess the speed of the host computer system 
(step 63). All synchronization variables are output to global data buffers to be utilized by 
time dependent subroutines, array determinant threshold subroutines, and polling count 
determinate subroutines within the simulation program. Next, the program proceeds to a 
case presentation module 47. 

In case presentation module 47, a patient case and its relative data are retrieved (step 
65) via a random selection subroutine 67, from an on-line database 69. This stage may be 
replaced or supplemented by a routine allowing the user to select a specific patient case 
from the case files, or to input data from a selected patient case not in the stored files. The 
latter would be particularly useful for conducting a preoperative rehearsal. The case 
specific data which is stored in memory in the preferred embodiment can include EKG data 
(normal and abnormal), occlusion location data and angioplasty success data, i.e., the 
required number of successive balloon inflation cycles required to remove the occlusion. 
(The angioplasty success data is artificial since in an actual procedure the number of balloon 
inflations required to remove an occlusion typically is not known in advance.) The case 
determinant variables are set in step 71. 

Patient information is delivered via a multimedia presentation at step 12. This may 
include a patient history screen as shown in Fig. 18, an arteriogram as shown in Fig. 19, 
and an indication of hemodynamics as seen in Fig. 20. In addition to displaying patient 
information on the screen, corresponding or supplemental information is conveyed by a 
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digitally recorded voice provided through the multi-media audio system. If footswitch 13 
is activated at step 75, the presentation will proceed ("fast forward") to the next topic (step 
77). So long as the program polling does not detect an activated footswitch, the program 
will proceed in a linear fashion through each topic, until the last topic is reached (step 79). 

Before the simulation actually begins, the catheter interface device is calibrated 
within calibration module 49 (step 81). To calibrate catheter interface device 3, the 
catheter wire 7 is synchronized with the program by the user. The user is instructed by the 
program to pull the guide wire out while a status bar on the screen display of monitor 10 
decreases to indicate the approach of the catheter starting point. This calibration is required 
for the program to ascertain the point of origin for screen representation and translation 
tracking. Rotation degree is also set to a starting position at this time. 

Referring now to Fig. 4, upon completion of the calibration routine, the program 
will proceed to a catheter placement simulation module 83. At this stage, the screen 
displays a simulated fluoroscopic (X-ray) view (see Fig. 22), wherein the patient's rib-cage 
84 is clearly seen. The arterial tree remains invisible until a subsequent dye infusion 
routine is executed. Once the dye infusion routine is executed, the arterial tree 86 including 
an occluded region 88, and the virtual catheter 90, become clearly visible (see Figs. 23-29). 
The background image can be created by digitizing an actual fluoroscopic image and, if 
desired for clarity, by modifying the digitized image data to remove unwanted structures 
and "noise" . In the present embodiment, this background remains inanimate throughout the 
simulation (as it generally would during an actual procedure). As also seen in Figs. 21-29, 
the screen presents along its bottom border a moving EKG waveform 92, based on the 
stored patient data. 

In catheter placement simulation module 83, a timer subroutine 85 simulates real 
time by counting units that are determined by the polling rate time adjustment step 63. 
Referring to Fig. 5 showing the timer subroutine, the timer unit is defined as an integer 
with a value of 1. If the current sum timer value is greater than 1 at step 85a, then the 
value is increased by 1 unit at step 85b. The value is set to 1 upon initiation of the timer, 
at step 85c. 

Referring back to Fig. 4, the program proceeds to an EKG data determination 
subroutine 87. Referring to Fig. 6, if an occlusion (blockage) is indicated in the case 
specific arterial dataset (blockage =1) at step 87a then the program will check to see that 
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the abnormal EKG data is stored in the program memory buffer (step 87b). If it is not yet 
stored, the case determinant data is retrieved at step 87c from the EKG database 87d. The 
EKG data base 87d can provide sinus rhythm patterns as well as atrial fibrillation data, and 
other case determinate cardiac events. The buffer space is given a local definition, i.e. 
CASEAJBNORMAL at step 87e, and the EKG data is placed in a common buffer (step 
87f). Alternately if the correct buffer name is found at step 87b, then the program will 
proceed out of the subroutine. 

Conversely, if it is determined that a blockage does not exist at step 87a, then the 
program will check to see that the EKG data is set to CASE_NORMAL at step 87g. If it 
is, the program will proceed out of the subroutine. Otherwise, the CASE_NORMAL data 
is retrieved at step 87h from the EKG database 87d. The buffer name is set to 
CASE_NORMAL at step 87i, and the data in the EKG data buffer is overwritten at step 
87f. The program provides the onscreen EKG simulation based on this data in the EKG 
Buffer, which is derived from the stored case data. 

Referring momentarily back to Fig. 4, the onscreen EKG display is produced and 
updated within subroutine 89. Referring to Fig. 7, the update EKG subroutine will check 
to see that the EKG data is translated into a usable array at step 89a. If an array is not 
detected, the EKG data is retrieved at step 89b from the EKG data buffer 89c and sorted 
into an array. The array is processed at step 89d by translating the value from the EKG 
buffer array 89c, to a floating-point number between 0 and 1. A value equal to 75% to 
80% of the highest value in the translated array is set as an audible beep threshold value 
at step 89e. The adjusted array is next defined as the spline deformation array at step 89f . 
If it is determined at step 89a that the spline deformation array has already been set, the 
subroutine will bypass the array definition process. 

Each time the program polls update EKG screen subroutine 89, the spline 
deformation array is advanced one place holder (step 89g). A range from the array is then 
used to deform the EKG screen spline at step 89h. The real patient data may be plotted and 
the EKG screen decay can be simulated. If it is determined at step 89i that the apex of the 
spline is at or above the defined beep threshold value, then an audio beep is generated by 
the sound system 89j, at step 89k. 

The simulation can include, as a simulated complication, randomly generated periods 
of EKG irregularity, e.g., atrial fibrillation. This is illustrated in the screen display of Fig. 

- 13 - 



SUBSTITUTE SHEET (RULE 26) 



WO 96/28800 



PCT/US96/02317 



21, wherein the surgeon is instructed to stand-by until normal sinus rhythm resumes. At 
this point in an actual procedure, a drug would be administered to the patient. Such a step 
could be included in the present simulation, e.g., using footswitch 13 as a trigger 
mechanism. Alternatively, the simulation can be simplified by simply notifying the surgeon 
that the drug has been administered. Another feature that can be provided is a graphic time 
clock 94 (as shown in Fig. 21) notifying the surgeon of the time remaining before the drug 
takes effect (and sinus rhythm is resumed). 

Referring again to the main program of Fig. 4, at subroutine 91 an arterial pulse is 
realistically simulated by synchronizing arterial and catheter movement with the EKG pulse 
data. Referring to Fig. 8, subroutine 91 will retrieve a spline deformation array at step 
91a, from a spline deformation buffer 91b. The spline deformation array is the data used 
to represent the virtual arterial tree that will appear upon dye infusion. The configuration 
of the arterial tree can be adapted from commercially available 3-D graphics packages (such 
as from ViewPoint Data Labs of Orem, Utah) or can be independently generated based 
upon physical models or actual anatomies. Offset data points are extrapolated from the 
array data at step 91c. The geometry is offset (redrawn) and the screen is updated at step 
91d. The program then proceeds to step 93 of the main program (Fig. 4), where the status 
of footswitch 13 is checked. At this stage, footswitch 13 is used to initiate the infusion of 
contrasting dye (subroutine 95). The default position of the switch is OFF and the program 
will simulate rapid dissipation of the dye into the bloodstream (as would occur in an actual 
procedure) when the footswitch is released. At step 93, a check is made to see whether the 
footswitch has been depressed. If not, dye infusion subroutine 95 is by-passed. 

When the footswitch is depressed, the program proceeds to the dye infusion 
subroutine 95. Referring to Fig. 9, dye infusion is activated at step 95a, whereupon a 
ramp constrained, random flow of values is streamed into an infusion value definition (see 
95b) used to determine the shade or concentration of the dye. By altering the value of the 
dye concentration at various points, the shade variations simulate the mixing of blood and 
dye. By ramping the value stream, simulated dye that is infused behind dye infused earlier 
will appear less diffused, thereby looking to be of higher concentration. By randomizing 
the shade determinative value stream as it is ramped upwardly, it is possible to create a 
strikingly realistic simulation of one fluid (in this case dye) diffusing into another fluid (in 
this case blood). The constraints on the ramped random value stream 95b can be adjusted 
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to provide the appearance of different fluid viscosities and rates of diffusion. Alternatively, 
a particle system or procedural material map may be used. 

The path of the dye is mapped (at step 95c) in accordance with the artery data set 
95d. The flow of the dye is mapped progressively from the top of the artery down through 
the branches. When the footswitch is released, the mapped contrasting dye is reverse 
mapped to simulate diffusion in a downward direction, as if blood were flowing from above 
and "washing" the dye from the site. 

If it has been determined at step 95e that a puncture in the arterial wall has been 
mapped on the arterial geometry, and footswitch 13 is in the ON position, the puncture 
origin 95f is mapped at step 95g, to simulate bleeding from the puncture into the 
surrounding body cavity. Such bleeding is shown at 96 in the screen display of Fig. 25. 
If no origin is detected, the program will proceed. As will be described, a puncture is 
caused to be mapped onto the geometry when the user moves the catheter wire 7 of the 
catheter interface device 3 too abruptly. 

The next step in the main program is a catheter movement sampling subroutine 97. 
Catheter interface device 7 includes an electronic interface having a software driver for 
interpreting the signals produced by the catheter wire tracking sensors. As shown in Fig. 
10, the data that is provided by the driver is sampled at step 97a, at a rate that has been 
determined by the program at step 63. Longitudinal translation 97b and rotation 97c of the 
wire through the catheter interface device are processed to determine the wire location 
relative to the calibration point position established in step 81. The Z-axis (corresponding 
to the catheter insertion depth) and the rotation data are placed in an array buffer in step 
97d. 

The program then proceeds to subroutine 99 for determining the location and 
orientation of the virtual catheter. As shown in Fig. 1 1 , this is computed at step 99a from 
the Z-axis and rotation value array stored in array buffer 97d, a geometry offset value 99b, 
and the constraints provided by the artery data set 99b. By taking account of a geometry 
offset value (which is a product of the arterial geometry data and the EKG data), a 
simulated flutter or vibration of the catheter in synchronism with the arterial pulse is 
achieved. The catheter location and orientation data are placed in a data buffer at step 99d 
for use in a subsequent catheter update subroutine. 
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With the arterial tree illuminated by simulated fluoroscopic dye, the catheter is 
navigated through the arteries toward the occlusion region. Figs. 23-26 show the 
progression of virtual catheter 90 through arterial tree 86 and into occluded region 88. As 
can be seen in these screen displays, virtual catheter 90 has an angled or bent end portion 
98. This allows for steering the catheter down different pathways, e.g., branches in the 
virtual arterial tree, based upon the rotational orientation of the catheter wire. 

To determine whether the catheter has entered occluded region 88, within subroutine 
101 it is determined if the spline origin (representing the catheter end) is within the 
blockage range stored as part of the case specific data. Referring to Fig. 12, the blockage 
range data 101a, and catheter location definition 101b are compared at step 101c. A local 
value of 1 is output at step lOld if the comparison determines that the catheter location is 
within the blockage range. Conversely, if the comparison indicates that the catheter is not 
within the occlusion region, a value of 0 is passed at step lOle. 

If subroutine 101 returns a value of 1, then the program will output a signal to the 
catheter interface device to impart force feedback (a.k.a. haptic feedback, resistive 
feedback) to the catheter wire (at step 103). The force feedback signal can be provided at 
a single preset level, or could be provided in gradations dependent on the location of the 
catheter and the cross-sectional dimensions of the occluded region. Then, the program 
proceeds to a balloon inflation simulation module 105. If subroutine 101 returns a value 
of 0, the program will proceed to subroutine 107 for checking whether the catheter has 
punctured the arterial wall. 

Puncture simulation in the illustrated program is based on a detected velocity of 
catheter wire 7 within the catheter interface device 3. This provides a realistic simulation 
in that during an actual procedure an abrupt movement of the catheter can often result in 
an arterial puncture. Likewise, in the simulation, if the catheter wire is moved abruptly by 
the user, the velocity of the catheter exceeds a preset level and thereby triggers the 
simulation of a puncture. 

Referring to Fig. 13, in puncture detecting subroutine 107, the program will first 
check at step 107a for the presence of a previously defined puncture origin 107b. If a 
puncture origin 107b does exist, then the program checks at step 107c whether an audio 
response (e.g., verbal announcement of the error) through the multimedia sound system 
107d is completed. During the audio response, the program continues to loop, resulting 

- 16- 



SUBSTITUTE SHEET (RULE 26} 



WO 96/28800 



PCI7US96/02317 



in the simulation of bleeding from the puncture origin (see 96 in Fig. 25), due to execution 
of dye infusion subroutine 95. At this time, a flashing screen message may also appear to 
alert the user of the procedural mistake. Following completion of the audio response, the 
simulation proceeds to termination step 107e, whereupon the screen display can be frozen 
to preserve the unsuccessful result for evaluation. 

If a puncture origin is not detected at step 107a, the presence of a previously logged 
velocity variable is determined at step 107f. If none is detected, the current time 107g is 
placed in a local variable definition (buffer A) at step 107h and the program proceeds. If 
a previously recorded value is detected in step 107f, then that variable is moved to a second 
buffer (buffer B), in step 107i. The current time 107g is placed in the primary buffer (A) 
at step 107j. The time between events is then calculated at step 107k. If the calculated 
elapsed time is determined to be below a preset value at step 107e, this indicates a velocity 
in excess of the velocity constraint. Under this condition, the puncture origin 107b is set 
to the current catheter location (step 107m). The program will then activate the audio 
response to the puncture event (step 107n), and program control returns to the main 
program. If the time between events calculated in step 107k is found (in step 107e) to be 
within the defined constraints, the program will proceeds to the next stage without a 
puncture origin having been defined. 

An alternative method of detecting a puncture would be to utilize a collision 
detecting component, whereby a puncture would be simulated upon an impact of the virtual 
catheter tip against a wall of the virtual artery. 

In the next main program step (subroutine 1 1 1) , the catheter position and orientation 
are updated. The catheter is represented by a spline generated from data points. Referring 
to Fig. 15, to determine the length of the spline, the starting point (i.e. catheter origin 
111a), and the current location of the catheter 111b, are retrieved and processed in step 
111c. In step Hid, the spline length definition computed in step 111c is combined with 
geometry offset data llle. The curvature of the spline (i.e., spline deformation) is derived 
from the geometry offset data 1 1 Id, which is a product of the artery geometry and the EKG 
pulse information. If a puncture origin is present (1 1 If), the path of the spline will not be 
constrained by the geometry map at that point, thus allowing the catheter to pass through 
the puncture, e.g., as seen at 96 in the screen display of Fig. 25. 
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The rotation of the catheter wire is reflected by the position of the bent or angled 
wire end portion of the virtual catheter. The catheter is pointed or navigated on the path 
chosen by the user by rotating the wire (and bent end portion thereof). If it is determined 
in step lllg that the end of the catheter is located at a branch in the arterial data set lllh 
(as seen at 100 in the screen display of Fig. 24), the program will determine the direction 
of travel or trajectory at step llli based upon retrieved rotation data lllj. Finally, the 
spline is deformed and redrawn in step 1 1 lk, and program control proceeds. 

Referring back to Fig. 4, when the catheter is determined to be within the occlusion 
range at step 101, and after force-feedback is activated at step 103, the program proceeds 
to balloon inflation module 105. Therein, the program will first magnify the occlusion 
region and change the viewing perspective (step 113), as seen in the screen displays of 
Figs. 27-29, in order to enhance visualization of the arterial structure and the effect of the 
procedure thereon. Contrasting dye is automatically activated, the visual components are 
adjusted to the new perspective, and the polling cycle is redirected to loop within the 
balloon inflation module. The required program components are replicated within this 
stage, i.e., catheter update 111, timer 85, EKG data 87, arterial pulse update 91, and 
footswitch status check 93. Within the balloon inflation module, footswitch 13 becomes 
operative to activate balloon inflation rather than dye infusion. If step 93 determines that 
the footswitch is activated, the program proceeds to a balloon inflation subroutine 115. 

Referring to Fig. 14 showing balloon inflation subroutine 115, if the footswitch is 
depressed so that a value of 1 is detected at step 115a, then the program proceeds to step 
115b wherein it is determined whether an inflation status buffer 115c has been previously 
established. If the buffer has been established, the value within the buffer is increased by 
one unit, at step 115d. The value from the inflation status buffer is then used to deform 
(inflate) the balloon tip 102 of the virtual catheter (step 115e) and then to correspondingly 
deform the occlusion geometry outwardly (step 1 150- These effects are seen clearly in the 
screen display of Fig. 28. 

If it is determined at step 115b that an inflation status buffer has not yet been 
previously established, then one is created at step 1 15g. The value is set to one unit at step 
115h, and the value from inflation status buffer is then used to deform (inflate) the balloon 
tip of the catheter (step 115e), and then to correspondingly deform the occlusion geometry 
outwardly (step 115f). 
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If the footswitch value is not equal to 1 at the time step 115a is polled, then the 
program will determine at step 115i if an inflation status buffer 115j has been previously 
established. If it has not, the program proceeds to loop within the main balloon inflation 
module 105. If it has, the buffer is reduced by 1 unit at step 115k, thus serving to 
decrementally deflate the balloon catheter. If the buffer is reduced to a value of 0 (step 
1151), effectively deflating the balloon (as seen in Fig. 29), the buffer is released from 
memory at step 115m. If the condition exists that the buffer has been reduced by 1 unit 
(step 115k) but has not been reduced to a value of 0, then the decremented value from the 
inflation status buffer is then used to deform (in this case deflate) the balloon tip of the 
catheter, and then to correspondingly deform the occlusion geometry, at steps 115e and 
115f . 

On exiting balloon inflation subroutine 115, program control proceeds to a balloon 
rupture subroutine 117. A threshold value for inflation is set upon the case specific arterial 
dataset retrieval. This threshold should be equal to the value of units required to fully 
deform the blocked region of the artery so that it is of healthy dimensions. It is possible 
also to introduce the threshold at a lower value to provide for a risk management 
educational element to the program. Referring to Fig. 16, if the value retrieved from the 
inflation status buffer 117a is determined to exceed the threshold value at step 117b, then 
the balloon volume is set to 0 units (step 117c), abruptly deflating the balloon (as seen in 
Fig. 29) and thereby simulating balloon rupture. This action can be buffered over time to 
deflate at a visually correct rate. Various events such as fragmentation and vascular 
damage may be added to this scenario to enhance the program's realism. When a rupture 
has occurred, the rupture audio response 117d is activated at step 117e, and the program 
then proceeds to a final summation of the simulation step 117f. 

The desired outcome (objective) of the simulation is clearance of the occlusion and 
restoration of normal sinus rhythm. If balloon rupture is avoided in subroutine 117, 
process control proceeds to subroutine 119 for determining when the occlusion has been 
successfully cleared. Referring to Fig. 17, the program will detect whether the balloon has 
been allowed to inflate sufficiently to cause complete deformation of the occlusion range 
cross-sections to the proper (corrected) diameter. If so, an occlusion value of 0 will be 
detected at step 1 19a. If the cross-sections have not been fully corrected, then the program 
will continue to loop. 
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In an actual procedure, removal of a blockage typically requires repeated balloon 
inflation cycles. Similarly, in the simulation, repeated inflation cycles are required to 
successfully clear the occlusion. The required number of successive inflation cycles is 
determined by a success value definition which is part of the retrieved case specific data. 
For each polling cycle wherein it is determined that the blockage has been fully deformed 
by the balloon to the corrected cross-section, a count value is incremented at step 119c and 
a comparison is made at step 1 19d to see if the success value definition has been reached. 
If so, the program proceeds to a successful concluding summation at step 119e. The 
concluding summation may, e.g., include an announcement of the successful result, and 
provide a display of the elapsed time of the procedure. If the success value has not been 
reached, the program continues through the polling loop. Importantly, because of the 
inflation time limitation imposed by the balloon rupture subroutine 117, a user cannot 
achieve a successful clearance of the occlusion by simply maintaining the footswitch in the 
depressed (ON) position. Rather, it is necessary for the user to successively inflate the 
balloon and allow it to deflate (via the decrementing function of step 115k) at least several 
times, as in an actual procedure. 

In the preferred embodiment, the software is programmed using the industry 
standard language C++ and OpenGL. The use of these two languages has several 
advantages: 

• ability to port the software to different hardware platforms. OpenGL has 
been licensed by many different workstation manufacturers including 
Windows NT, ATT, DEC, and DOS. OpenGL is an open, documented 
standard for describing 3- dimensional interactions with objects. 

• ability of software to adapt to upcoming hardware innovations. OpenGL 
handles most graphics routines, and its use is scaled to the particular 
hardware that is available on the computer. 

• compatibility with Nintendo/Silicon Graphics, Inc. hardware arrangements, 
as well as set-top boxes intended for interactive television. The availability 
of inexpensive platforms for VR will open opportunities for use in third 
world countries, rural settings and individual homes and offices. 

In order to maintain real-time frame rates (30 fps), it is important to limit picture 
detail to the essential portions of the screen. The limited resource is the graphics rendering 
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speed (the number of polygons per second), and the software must know where to best 
allocate this resource to determine what features being displayed should have the most 
detail. In the exemplary cardiac catheterization simulation described herein, the real time 
graphics are limited to the arterial tree and the catheter therein; the remaining background 
of the simulated fluoroscopic view is inanimate (as it would be in an actual procedure). 
With this arrangement, real time rates are readily attainable. 

Specific attention will now be given to the preferred system hardware for the above- 
described cardiac catheterization simulation system. There are a number of different 
hardware platforms that can be utilized for VR applications. Current configurations 
include: 

High end systems . VR can be run on systems such as the Silicon Graphics, Inc. 
RealityEngine. These are costly and logistically the most difficult to transport due to their 
large size. The Silicon Graphics Indigo 2 Extreme workstation has the graphics and 
computer power necessary for many VR applications, and is less costly. 

Personal cnmputer systems . Although the current generation of personal computers 
could be used for some limited VR applications, they lack the specialized hardware found 
in graphics workstations which facilitates the production of real-time complex 3-D 
manipulation of objects. Personal computers are optimized for 2-D graphics, and as a 
result, they are ideal for playback of digitized linear sequences which can be obtained from 
3-D graphics systems (VR). Personal computers are also well suited for the display of 2-D 
representations of 3-D phenomena. 

The angioplasty application of the present invention was developed on a Silicon 
Graphics Indigo 2 Extreme workstation. A complete hardware specification for this 
application is set forth below: 

• Silicon Graphics Indigo 2 Extreme graphic hardware (MIPS R4000/100 
Mhz, 64MB RAM, 1GB hard drive, 19" SGI monitor, CD ROM drive, 
DAT tape drive. 

• stereographic equipment (e.g., CrystalEyes eyeglasses and emitter + cable). 

• audio equipment (one pair of stereo power speakers, Mackie 1202 (or 
compatible) mixer or another pair of power speakers. 

• audio CD for providing continuous background music (narrative "voice- 
over" can be recorded onto hard drive). 
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• miscellaneous (two stereo mini-male to RCA male cables, six RCA female 
to 1/4 M male phone jack, power strip). 

• catheter interface device (as described in detail hereinbelow). 

• foot pedal switch peripheral, e.g., as is available from Immersion 
Corporation. 

In the preferred embodiment, the catheterization simulation system of the present 
invention employs a virtual catheter interface produced for use in applicants' system by 
Immersion Corporation. This device is now described in detail with references to Figs. 30- 
37. 

Catheter interface device 3 tracks the movement of wire 7 in a 3-D space, with the 
movement of the wire constrained to two degrees of freedom. This provides a realistic 
simulation of an actual catheter insertion procedure, wherein once the catheter is inserted 
into a patient, it is limited to two degrees of freedom. More particularly, within a patient's 
body, the catheter wire is constrained to translation (in-out motion) and rotation. 

The catheter wire may be formed of a variety of known materials, e.g., metal or 
nylon. Generally, the gauge should not be bigger than 050 (0.050* diameter), and the 
insertion length should be at least 14". The catheter wire can be an actual catheter wire, 
such as is commercially available, e.g., from Cordis Corporation of Miami Lakes, Florida. 
An actual catheter should be modified such that any tool at the end (such as any cutting 
edges or balloon) is removed leaving only the handle and the shaft. 

Catheter wire 7 can include an end mounted handle or "grip" portion 121. The 
"grip" portion can be any conventional device used to manipulate a catheter. Alternatively, 
the catheter shaft could be gripped directly. While the particular embodiment described 
herein comprises a wire, it should be understood that other catheters and catheter-like 
members could be used as well, including particularly elongated flexible tubular catheters. 

An electronic interface 123 serves to electrically couple the force feedback actuator 
and motion sensors contained in catheter interface device 7 to the computer 5. An 
electronic interface that is particularly adapted for the present invention is commercially 
available from Immersion Corporation. This is basically the same electronic interface 
designed for use with Immersion's PROBE. Unlike the PROBE setup, a serial interface 
is configured to be bi-directional, so as to allow for transmission of a tactile feedback signal 
from the computer to the force feedback actuator. The electronic interface has six channels 
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corresponding to the six degrees of freedom of the Immersion PROBE. However, with the 
present invention, the electronic interface requires use of only two channels for the tracking 
of catheter wire 7, since the catheter wire is constrained to two degrees of freedom. A 
third channel is used for transmission of the tactile feedback signal. 

The electronic interface includes a software driver. The update speed of the device 
is between 500Hz and 2kHz - this speed cannot be exceeded with a single serial port. If 
necessary to achieve a cleaner "feel," the device can be connected with two serial ports 
running simultaneously, whereby a 2X speed improvement is obtained by doubling the data 
transmission capability. Other methods of improving interface speeds between the catheter 
interface device and the computer include standard and high speed ethernet connections, use 
of SCSI or direct memory access parts (DMA), or a variety of more rapid standard 
interfaces. In addition, dedicated hardware to facilitate input/output can be used, such as 
boards for rapid data acquisition. 

The operative components of catheter interface device 3 are mounted on top of the 
housing of electronic interface 123 within a separate metal housing 125 having a removable 
cover 127. A rubber-like tubular member 129 is used to assure free passage of wire 7 
through the housing cover 127. Housing 125 could instead be modeled after the external 
bodypart(s) located within the vicinity of the catheter insertion location, e.g., a leg or arm, 
in order to render the simulation components more "transparent" to the user. 

The operative components of catheter interface device 3 comprise a block assembly 
131 forming a through-passage 133 (see Fig. 37) and mounting locations for a first input 
transducer (sensor) 135 that senses the translational movement of wire 7 and produces a 
corresponding electrical signal for input to computer 5, an output transducer (actuator) 137 
that imparts a resistance force to wire 7 responsive to a tactile feedback signal received 
from computer 5, and a second sensor - a rotational transducer - 139 for tracking the 
rotation of wire 7. 

Block assembly 131 comprises a plurality of blocks formed of aluminum or other 
lightweight material, such as plastic. The blocks may be cast, molded, and/or machined 
as a monoblock members, with the aforementioned sensors and actuator attached thereto 
and/or incorporated therein. 

A variety of transducers, readily available in the commercial market are suitable for 
use in the present invention. For example, the input transducers (sensors) can include 
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encoded wheel transducers, potentiometers, optical encoders, etc. The output transducer 
(actuator) could be a stepper motor, servo motor, magnetic particle brake, friction brake, 
pneumatic actuator, etc. Hybrid or bi-directional transducers that pair input and output 
transducers together could also be used in place of separate sensor and actuator devices. 
A simple bi-directional transducer that could be used is a permanent magnet electric 
motor/generator. 

It should also be noted that the present invention can utilize both absolute and 
relative sensors. An absolute sensor is one in which the angle of the sensor is known in 
absolute terms, such as with an analog potentiometer. Relative sensors only provide 
relative angle information, and thus require an initial calibration step, as is provided in the 
present simulation system. 

In the preferred embodiment, actuator 137 is a solenoid capable of producing a force 
output of at least 5 ounces for imparting tactile feedback in the form of passive resistance 
(factional drag) on wire 7. As best seen in Figs. 35 and 37, a solenoid plunger 141 has 
attached to its end a lower pressing plate 143. Lower pressing plate 143 engages wire 7 
against an upper pressing plate 145 by applying a force in a direction substantially 
perpendicular to the translation direction of wire 7. This produces a frictional force along 
the translation direction of wire 7. Such solenoids, controllable to a thousand or more 
selectable resistance force values, are available commercially from, e.g. , Guardian Electric 
of Woodstock, Illinois. It will be appreciated that other actuator devices may be employed 
in the present invention. For example, the actuator could be a pneumatic or hydraulic 
device which applies a force to the wire. It will be appreciated by those skilled in the art 
that the choice of electromechanical, electromagnetic, pneumatic or hydraulic actuation will 
depend largely on the required response time, and the cost and complexity of the device. 
Electromechanical/electromagnetic transducers are generally preferred as they typically have 
a fast response time, are low cost, and are simpler and smaller than hydraulic and 
pneumatic devices. 

One or both of upper and lower press plates 143, 145 can comprise a resilient 
friction (brake) pad 147 (shown attached to upper press plate 145), for providing a smooth 
braking action to wire 7. Materials appropriate for the resilient brake pad will be apparent 
to those of ordinary skill in the art. 
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As best seen in Figs. 36 and 37, the translation sensor incorporates a friction wheel 
149, which wheel is mounted on a rotatable shaft 151. The friction wheel engages wire 7 
with a normal force such that translation of wire 7 causes rotation of shaft 151. As best 
seen in Fig. 35, shaft 151 is coupled to a disk 153 made of clear plastic material and having 
a number of dark radial bands 155 formed on its circumference. As best seen in Fig. 36, 
a photodetector pair 157a,b including a light source and a detector is positioned on opposite 
sides of disk 153 in alignment with bands 155. As disk 153 rotates about its central axis, 
bands 155 alternatingly cause light to impinge and not impinge upon the detector, thereby 
creating an electronic signal which is passed to electronic interface device 123 over a bus 
wire 157. In the illustrated preferred embodiment of catheter interface device 3, 
translational movement of the wire is tracked with a resolution of at least 0.002". One 
suitable type of sensor is an optical encoder wheel type input transducer (Model SI) 
marketed by U.S. Digital of Vancouver, Washington. The same basic type of sensor can 
be used for rotation sensor 139. 

In the preferred embodiment, the rotation sensor 139 includes a disk 159 rotatably 
coupled with wire 7. Disk 159 is preferably made from a clear plastic material and has a 
number of dark radial bands 160 formed on its circumference. A photodetector pair 161 
including a light source and a detector is positioned on opposite sides of disk 159 in 
alignment with bands 160. As disk 159 rotates about its central axis, the bands 
alternatingly allow light emanating from the light source to impinge and not impinge upon 
the detector, thus producing an electronic signal which is also passed to electronic interface 
device 123 over bus wire 157. A second photodetector could be provided so as to allow 
determination of the rotation direction, as is well known to those skilled in the art. 

The engagement between wire 7 and the central aperture of disk 159 is preferably 
accomplished by the formation of a friction seal. This can be provided by a resilient (e.g. , 
rubber or plastic) grommet 163 mounted within the hub of disk 159. Grommet 163 
receives wire 7 with a semi-tight fit causing disk 159 to rotate coaxially with the wire, 
while at the same time allowing relatively free wire translation. Preferably, the mass of 
the rotating transducer parts is kept very small so that it only takes a small amount of 
friction to ensure coaxial rotation of wire 7 and disk 159 without slippage. Because the 
level of friction is so small, it does not substantially impede translational motion (i.e., in- 
out motion) of wire 7. Alternatively, wire 7 could include a lengthwise extending rib, slot 
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or flat, which is engaged with a corresponding key- way provided on disk 159. While more 
costly and difficult to manufacture, the use of a non-circular wire and corresponding key- 
way would ensure a slip-free connection between the wire and the disk, without introducing 
translational friction. 

The principles of the present invention can also be applied to create an intravenous 
(TV) line insertion simulator. In the IV insertion simulator, a hardware device and software 
work together to reproduce the visual and physical experience of inserting an IV needle into 
an artery. The simulator would work by having the user grasp a proxy needle connected 
to a position monitoring device. Preferably, a custom IV hardware interface device would 
be designed for this purpose. Alternatively, the Immersion PROBE could be used for 
performing a general simulation (without tactile feedback). The insertion would be carried 
out by interacting with a simulation of the activity on a computer generated display. The 
display would show the needle as the user moves it along the part of the body into which 
the needle is to be inserted. As the needle is pushed into the skin, the simulation software 
would detect the contact and display an appropriate view of the process. For instance, if 
the needle were properly inserted into an artery, then a small amount of bleeding could be 
displayed along with a change in the shape of the skin where the needle has passed under. 
In addition, a force mimicking that encountered by the user during an actual IV insertion 
could be generated by the IV interface device, in response to commands from the simulation 
program, similar to the arrangement cardiac catheterization simulator described in detail 
herein. 

The concepts of the present invention are applicable to provide a realistic simulation 
of a wide range of other medical procedures, including particularly invasive surgical 
procedures wherein a device is inserted into an internal body passageway, or directly into 
body tissues. The procedures which can be simulated using the present inventive techniques 
span every specialty of medicine, including but not limited to cardiology, invasive 
radiology, neurosurgery, neurology, anesthesiology and, of course, general surgery. 
Procedures that can be simulated with the inventive system include, but are not limited to: 

• placement and/or use of therapeutic mechanical devices, e.g., angioplasty, 
stent placement and coil placement for aneurism treatment. 

• intravenous (TV) line and central line placement, e.g., jugular, femoral. 
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• introduction into the body of materials or devices to enhance the ability to 
discern internal body parts, e.g., angiography, intra-coronary ultrasound 
devices, and fiber-optic scopes. 

• placement of devices to obtain tissue and fluid samples from the body, e.g., 
lumbar puncture, and liver and kidney biopsy. 

• needle placement, e.g., epidural, phlebotomy simulation,deployment of 
therapeutic agents at the cite of injury/disease (e.g., use of a catheter for 
precise placement of chemotherapeutic agents at the cite of an intravascular 
tumor). 

• placement and manipulation of devices to remove fluids or tissues, e.g., 
urinary catheters, and pleural fluid removal. 

• devices for trauma management, e.g., chest tube placement. 

• devices for airway management, e.g., fiber-optic laryngoscope, rigid 
laryngoscope, and intubation procedures. 

• devices for assessing disease and pathology, e.g., fiber-optic scope 
manipulation (e.g., cystoscopes, bronchoscopy, flexible sigmoidoscopy, 
colonoscopy, and endoscopy). 

• devices for minimally invasive surgery, e.g., laparoscopic instrumentation. 
The present invention has been described in terms of preferred and exemplary 

embodiments thereof. Other embodiments, modifications and variations within the scope 
and spirit of the appended claims will, given the benefit of this disclosure, occur to persons 
of ordinary skill in the art. 
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CLAIMS 



1. A computer based catheterization simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 
body environment; 

second memory means for storing image data representing a virtual catheter 
extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 

first input means for receiving an output signal from a first computer 
peripheral device that tracks the translational movement of an elongated relatively flexible 
member, representing the virtual catheter, without a physical model corresponding to said 
virtual internal body environment; 

first calculating means for calculating position data representing the position 
of the virtual catheter within said virtual internal body environment, based on the signal 
from the peripheral device; and 

first updating means for updating the image data representing the virtual 
catheter extending within said virtual internal body environment, based on said calculated 
position data. 

2. A catheterization simulation system according to claim 1, wherein: 
the end of said virtual catheter is angled; 

said peripheral device tracks the rotational movement of the elongated 
relatively flexible member and outputs to the first input means a signal based thereon; 

said first calculating means calculates orientation data representing an 
orientation of the angled end of the virtual catheter, based on the signal from the first 
peripheral device; and said updating means updates the image data representing the virtual 
catheter based on the calculated orientation data. 

3. A catheterization simulation system according to claim 1 , further comprising: 
determining means for determining the location of the virtual medical 

instrument within the virtual internal body environment; and 

output means for outputting a tactile feedback response signal to said first 
computer peripheral device based on said location of the virtual catheter, said tactile 
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feedback response signal serving to cause a mechanism of said first computer peripheral 
device to impart to said elongated relatively flexible member a tactile feedback force. 

4. A catheterization simulation system according to claim 3, wherein said tactile 
feedback force is a force resisting movement of said elongated relatively flexible member. 

5 . A catheterization simulation system according to claim 1 , further comprising: 
second calculating means for calculating the velocity of the virtual catheter 

within the virtual internal body environment, based upon the signal from the peripheral 
device; and 

second updating means for modifying the image data representative of the 
virtual internal body environment, based on the calculated velocity data. 

6. A catheterization simulation system according to claim 5, wherein the second 
updating means modifies the image data representative of the virtual internal body 
environment when the calculated velocity of the virtual catheter exceeds a preset velocity. 

7. A surgical catheterization simulation system according to claim 6, wherein 
the second updating means modifies the image data representative of the virtual internal 
body environment to simulate a puncture of an internal body passageway, and associated 
bleeding. 

8. A catheterization simulation system according to claim 1, further comprising 
third memory means for storing image data representing a simulated real-time wave form 
of a signal representing a physiological parameter, and wherein said display means displays 
said simulated real-time wave form together with said images corresponding to said image 
data stored in said first and second memory means. 

9. A catheterization simulation system according to claim 8, wherein said 
simulated real-time waveform is an electrocardiogram (EKG) waveform. 

10. A catheterization simulation system according to claim 9, further comprising 
second updating means for modifying the image data representative of the virtual internal 
body environment, based on the image data representing the EKG waveform. 

11. A catheterization simulation system according to claim 10, wherein the 
virtual internal body environment comprises a vascular passageway, and the image data 
modification based on the simulated real-time waveform simulates a pulsation of the 
vascular passageway in synchronism with the EKG waveform. 

12. A catheterization simulation system according to claim 1 , further comprising: 
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a second input means for receiving a signal from a second computer 
peripheral device; and 

second updating means for modifying the image data representative of the 
virtual internal body environment, based on the signal from said second computer peripheral 
device. 

13. A catheterization simulation system according to claim 12, wherein the 
virtual internal body environment comprises a vascular passageway, and said image data 
modification based on the signal from the second computer peripheral device simulates the 
release and flow of contrasting dye through the vascular passageway. 

14. A catheterization simulation system according to claim 1, further comprising: 
a second input means for receiving a signal from a second computer 

peripheral device; and 

second updating means for modifying the image data representing the virtual 
catheter extending within said virtual internal body environment based on the signal from 
said second computer peripheral device. 

15. A catheterization simulation system according to claim 14, wherein the 
virtual internal body environment comprises a vascular passageway having an occlusion 
therein, and said image data modification based on the signal from the second computer 
peripheral device simulates the inflation of a balloon catheter within said occlusion. 

16. A catheterization simulation system according to claim 15, wherein an 
inflation amount of the simulated balloon catheter increases with time while a switch of the 
second computer peripheral device is held in an ON position. 

17. A catheterization simulation system according to claim 1 6, further comprising 
third updating means for modifying the image data representative of the virtual vascular 
passageway, based on the inflation amount of the simulated balloon catheter. 

18. A catheterization simulation system according to claim 17, wherein the image 
data modification based on the inflation amount of the simulated balloon catheter simulates 
the clearing of the occlusion. 

19. A catheterization simulation system according to claim 16, wherein the image 
data modification based on the signal from the second computer peripheral device simulates 
the rupture of the simulated balloon catheter in the event the switch is held in the ON 
position for a time period exceeding a preset time limit. 
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20. A catheterization simulation system according to claim 1 , wherein said image 
data representative of a virtual internal body environment, and said image data representing 
a virtual catheter, comprise 3-D image data. 

21. A catheterization simulation system according to claim 11, wherein said 
vascular passageway comprises an arterial tree. 

22. A catheterization simulation system according to claim 13, wherein said 
vascular passageway comprises an arterial tree. 

23. A catheterization simulation system according to claim 15, wherein said 
vascular passageway comprises an arterial tree. 

24. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 
body environment; 

second memory means for storing image data representing a virtual medical 
instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 

first input means for receiving an output signal from a first computer 
peripheral device that tracks the movement of a physical member representing the virtual 

medical instrument; 

first calculating means for calculating position data representing the position 
of the virtual medical instrument within said virtual internal body environment, based on 
the signal from the peripheral device; 

first updating means for updating the image data representing the virtual 
medical instrument extending within said virtual internal body environment, based on said 
calculated position data; 

second calculating means for calculating the velocity of the virtual medical 
instrument within the virtual internal body environment, based upon the signal from the 
peripheral device; and 

second updating means for modifying the image data representative of the 
virtual internal body environment, based on the calculated velocity data. 

25. A medical procedure simulation system according to claim 24, wherein the 
second updating means modifies the image data representative of the virtual internal body 
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environment when the calculated velocity of the virtual medical instrument exceeds a preset 
velocity. 

26. A medical procedure simulation system according to claim 25, wherein the 
second updating means modifies the image data representative of the virtual internal body 
environment to simulate a puncture of an internal body passageway, and associated 
bleeding. 

27. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 
body environment; 

second memory means for storing image data representing a virtual medical 
instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 

first input means for receiving an output signal from a first computer 
peripheral device that tracks the movement of a physical member representing the virtual 
medical instrument; 

first calculating means for calculating position data representing the position 
of the virtual medical instrument within said virtual internal body environment, based on 
the signal from the peripheral device; 

first updating means for updating the image data representing the virtual 
medical instrument extending within said virtual internal body environment, based on said 
calculated position data; and 

second updating means for modifying the image data representative of the 
virtual internal body environment, based on the signal from said peripheral device, to 
simulate a puncture of an internal body passageway, and associated bleeding. 

28. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 
body environment; 

second memory means for storing image data representing a virtual medical 
instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 

-32- 



SUBSTITUTE SHEET (RULE 26) 



WO 96/28800 



PCT/US96/02317 



first input means for receiving an output signal from a first computer 
peripheral device that tracks the movement of a physical member representing the virtual 
medical instrument; 

first calculating means for calculating position data representing the position 
of the virtual medical instrument within said virtual internal body environment, based on 
the signal from the peripheral device; 

first updating means for updating the image data representing the virtual 
medical instrument extending within said virtual internal body environment, based on said 
calculated position data; and 

third memory means for storing image data representing a simulated real-time 
wave form of a signal representing a physiological parameter; 

wherein said display means displays said simulated real-time wave form 
together with said images corresponding to said image data stored in said first and second 
memory means. 

29. A medical procedure simulation system according to claim 28, wherein said 
simulated real-time waveform is an electrocardiogram (EKG) waveform. 

30. A medical procedure simulation system according to claim 28, further 
comprising second updating means for modifying the image data representative of the 
virtual internal body environment, based on the image data representing the simulated real- 
time waveform. 

31. A medical procedure simulation system according to claim 29, further 
comprising second updating means for modifying the image data representative of the 
virtual internal body environment, based on the image data representing the simulated EKG 
waveform. 

32. A medical procedure simulation system according to claim 31, wherein the 
virtual internal body environment comprises a vascular passageway, and the image data 
modification based on the simulated real-time waveform simulates a pulsation of the 
vascular passageway in synchronism with the EKG waveform. 

33. A medical procedure simulation system according to claim 32, wherein said 
vascular passageway comprises an arterial tree. 



-33 - 



SUBSTITUTE SHEET (RULE 26\ 



WO 96/28800 



PCT/US96/02317 



34. A medical procedure simulation system according to claim 29, wherein the 
image data representing the EKG wave form includes EKG irregularity data for generating 
periods of EKG irregularity. 

35. A medical procedure simulation system according to claim 34, wherein said 
EKG irregularity is randomly generated. 

36. A medical procedure simulation system according to claim 28, further 
comprising second updating means for modifying the image data representing the simulated 
real-time waveform based upon a change in the image data representative of the virtual 
internal body environment. 

37. A medical procedure simulation system according to claim 29, further 
comprising second updating means for modifying the image data representing the EKG 
waveform based upon the existence or non-existence of data representing a simulated 
vascular abnormality in the image data representative of the virtual internal body 
environment. 

38. A medical procedure simulation system according to claim 37, wherein the 
simulated vascular abnormality comprises a vascular occlusion. 

39. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 

body environment; 

second memory means for storing image data representing a virtual medical 
instrument extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 

first input means for receiving an output signal from a first computer 
peripheral device that tracks the movement of a physical member representing the virtual 
medical instrument; 

first calculating means for calculating position data representing the position 
of the virtual medical instrument within said virtual internal body environment, based on 
the signal from the peripheral device; 

first updating means for updating the image data representing the virtual 
medical instrument extending within said virtual internal body environment, based on said 
calculated position data; and 
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second updating means for modifying the image data representative of the 
virtual internal body environment so as to simulate the infusion of one fluid into another. 

40. A medical procedure simulation system according to claim 39, wherein the 
simulated infusion of one fluid into another comprises simulated bleeding based on a 
calculated velocity of the virtual medical instrument. 

41. A medical procedure simulation system according to claim 40, wherein the 
velocity of the virtual medical instrument is calculated from said calculated position data. 

42. A computer based medical procedure simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 
body environment; 

display means for displaying images corresponding to said image data stored 
in said first memory means; and 

updating means for modifying the image data representative of the virtual 
internal body environment so as to simulate the infusion of one fluid into another. 

43. A medical procedure simulation system according to claim 42, wherein the 
simulated infusion of one fluid into another comprises a simulated infusion of contrasting 
dye into a bloodstream. 

44. A medical procedure simulation system according to claim 42, wherein the 
simulated infusion of one fluid into another is accomplished by streaming into a stored 
infusion value definition a ramp constrained random flow of values, to thereby vary the 
concentration of said one fluid at various points. 

45. A medical procedure simulation system according to claim 43, wherein the 
simulated infusion of contrasting dye into a bloodstream is accomplished by streaming into 
a stored infusion value definition a ramp constrained random flow of values, to thereby 
vary the concentration of the dye at various points. 

46. A computer based catheterization simulation system, comprising: 

first memory means for storing image data representative of a virtual internal 

body environment; 

second memory means for storing image data representing a virtual catheter 
extending within said internal body environment; 

display means for displaying images corresponding to said image data stored 
in said first and second memory means; 
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first input means for receiving an output signal from a first computer 
peripheral device that tracks the translational movement of an elongated relatively flexible 
member representing the virtual catheter; 

first calculating means for calculating position data representing the position 
of the virtual catheter within said virtual internal body environment, based on the signal 
from the peripheral device; 

first updating means for updating the image data representing the virtual 
catheter extending within said virtual internal body environment, based on said calculated 
position data; 

a second input means for receiving a signal from a second computer 
peripheral device; and 

second updating means for modifying the image data representing the virtual 
catheter extending within said virtual internal body environment based on the signal from 
said second computer peripheral device; 

wherein the virtual internal body environment comprises a vascular passageway 
having an occlusion therein, and said image data modification based on the signal from the 
second computer peripheral device simulates the inflation of a balloon catheter within said 
occlusion. 

47. A catheterization simulation system according to claim 46, wherein an 
inflation amount of the simulated balloon catheter increases with time while a switch of the 
second computer peripheral device is held in an ON position. 

48. A catheterization simulation system according to claim 47, further comprising 
third updating means for modifying the image data representative of the virtual vascular 
passageway, based on the inflation amount of the simulated balloon catheter. 

49. A catheterization simulation system according to claim 48, wherein the image 
data modification based on the inflation amount of the simulated balloon catheter simulates 
the clearing of the occlusion. 

50. A catheterization simulation system according to claim 47, wherein the image 
data modification based on the signal from the second computer peripheral device simulates 
the rupture of the simulated balloon catheter in the event the switch is held in the ON 
position for a time period exceeding a preset time limit. 
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